{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Metric Presentation and Visualization\n",
    "## Necessary packages and functions call\n",
    "\n",
    "- DDPM-TS: Interpretable Diffusion for Time Series Generation\n",
    "- Metrics: \n",
    "    - discriminative_metrics\n",
    "    - predictive_metrics\n",
    "    - visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'\n",
    "\n",
    "import sys\n",
    "sys.path.append(os.path.join(os.path.dirname('__file__'), '../'))\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "gpus = tf.config.experimental.list_physical_devices('GPU')\n",
    "if gpus:\n",
    "    try:\n",
    "        for gpu in gpus:\n",
    "            tf.config.experimental.set_memory_growth(gpu, True)\n",
    "    except RuntimeError as e:\n",
    "        print(e)\n",
    "\n",
    "from Utils.metric_utils import display_scores\n",
    "from Utils.discriminative_metric import discriminative_score_metrics\n",
    "from Utils.predictive_metric import predictive_score_metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Loading\n",
    "\n",
    "Load original dataset and preprocess the loaded data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "iterations = 5\n",
    "ori_data = np.load('../toy_exp/samples/sine_ground_truth_24_train.npy')\n",
    "# ori_data = np.load('../OUTPUT/{dataset_name}/samples/{dataset_name}_norm_truth_{seq_length}_train.npy')\n",
    "fake_data = np.load('../toy_exp/ddpm_fake_sines.npy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate the generated data\n",
    "\n",
    "### 1. Discriminative score\n",
    "\n",
    "To evaluate the classification accuracy between original and synthetic data using post-hoc RNN network. The output is | classification accuracy - 0.5 |.\n",
    "\n",
    "- metric_iteration: the number of iterations for metric computation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training: 100%|████████████████████████████████████████████████████████████████████| 2000/2000 [03:48<00:00,  8.77it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 0:  0.00649999999999995 , 0.5825 , 0.4305 \n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training: 100%|████████████████████████████████████████████████████████████████████| 2000/2000 [03:48<00:00,  8.75it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 1:  0.0034999999999999476 , 0.4425 , 0.5645 \n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training: 100%|████████████████████████████████████████████████████████████████████| 2000/2000 [03:32<00:00,  9.39it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 2:  0.0007500000000000284 , 0.46 , 0.5415 \n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training: 100%|████████████████████████████████████████████████████████████████████| 2000/2000 [03:46<00:00,  8.82it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 3:  0.02200000000000002 , 0.535 , 0.509 \n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training: 100%|████████████████████████████████████████████████████████████████████| 2000/2000 [03:48<00:00,  8.77it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 4:  0.007249999999999979 , 0.5365 , 0.478 \n",
      "\n",
      "sine:\n",
      "Final Score:  0.007999999999999985 ± 0.010231963047355045\n",
      "\n"
     ]
    }
   ],
   "source": [
    "discriminative_score = []\n",
    "\n",
    "for i in range(iterations):\n",
    "    temp_disc, fake_acc, real_acc = discriminative_score_metrics(ori_data[:], fake_data[:ori_data.shape[0]])\n",
    "    discriminative_score.append(temp_disc)\n",
    "    print(f'Iter {i}: ', temp_disc, ',', fake_acc, ',', real_acc, '\\n')\n",
    "      \n",
    "print('sine:')\n",
    "display_scores(discriminative_score)\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate the generated data\n",
    "\n",
    "### 2. Predictive score\n",
    "\n",
    "To evaluate the prediction performance on train on synthetic, test on real setting. More specifically, we use Post-hoc RNN architecture to predict one-step ahead and report the performance in terms of MAE. \n",
    "\n",
    "The model learns to predict the last dimension with one more step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training: 100%|████████████████████████████████████████████████████████████████████| 5000/5000 [04:59<00:00, 16.67it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0  epoch:  0.09314072894950631 \n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training: 100%|████████████████████████████████████████████████████████████████████| 5000/5000 [04:52<00:00, 17.08it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1  epoch:  0.09352861018187178 \n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training: 100%|████████████████████████████████████████████████████████████████████| 5000/5000 [05:04<00:00, 16.41it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2  epoch:  0.09322281220011404 \n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training: 100%|████████████████████████████████████████████████████████████████████| 5000/5000 [04:56<00:00, 16.85it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3  epoch:  0.09313142589665173 \n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training: 100%|████████████████████████████████████████████████████████████████████| 5000/5000 [05:01<00:00, 16.57it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4  epoch:  0.09342775384532898 \n",
      "\n",
      "sine:\n",
      "Final Score:  0.09329026621469458 ± 0.00022198749029314803\n",
      "\n"
     ]
    }
   ],
   "source": [
    "predictive_score = []\n",
    "for i in range(iterations):\n",
    "    temp_pred = predictive_score_metrics(ori_data, fake_data[:ori_data.shape[0]])\n",
    "    predictive_score.append(temp_pred)\n",
    "    print(i, ' epoch: ', temp_pred, '\\n')\n",
    "      \n",
    "print('sine:')\n",
    "display_scores(predictive_score)\n",
    "print()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Tensorflow_3.8",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
